Tootfinder

This https://arxiv.org/abs/2401.01750 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCV_…

Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement
The attention mechanism has been proven effective on various visual tasks in recent years. In the semantic segmentation task, the attention mechanism is applied in various methods, including the case of both Convolution Neural Networks (CNN) and Vision Transformer (ViT) as backbones. However, we observe that the attention mechanism is vulnerable to patch-based adversarial attacks. Through the analysis of the effective receptive field, we attribute it to the fact that the wide receptive field br…

MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning
Yichuan Li, Xiyao Ma, Sixing Lu, Kyumin Lee, Xiaohu Liu, Chenlei Guo
https://arxiv.org/abs/2403.06914

MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning
Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations). Nevertheless, the inclusion of demonstrations leads to a quadratic increase in the computational overhead of the self-attention mechanism. Existing solutions attempt to distill lengthy demonstrations into compact vectors. However, they often require task-specific retraining or compromise LL…

TREET: TRansfer Entropy Estimation via Transformer
Omer Luxembourg, Dor Tsur, Haim Permuter
https://arxiv.org/abs/2402.06919 https://…

TREET: TRansfer Entropy Estimation via Transformer
Transfer entropy (TE) is a measurement in information theory that reveals the directional flow of information between processes, providing valuable insights for a wide range of real-world applications. This work proposes Transfer Entropy Estimation via Transformers (TREET), a novel transformer-based approach for estimating the TE for stationary processes. The proposed approach employs Donsker-Vardhan (DV) representation to TE and leverages the attention mechanism for the task of neural estimati…

CSA-Trans: Code Structure Aware Transformer for AST
Saeyoon Oh, Shin Yoo
https://arxiv.org/abs/2404.05767 https://arxiv.org/pdf/2404.…

CSA-Trans: Code Structure Aware Transformer for AST
When applying the Transformer architecture to source code, designing a good self-attention mechanism is critical as it affects how node relationship is extracted from the Abstract Syntax Trees (ASTs) of the source code. We present Code Structure Aware Transformer (CSA-Trans), which uses Code Structure Embedder (CSE) to generate specific PE for each node in AST. CSE generates node Positional Encoding (PE) using disentangled attention. To further extend the self-attention capability, we adopt Sto…

This https://arxiv.org/abs/2403.18276 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csIR_…

RankMamba, Benchmarking Mamba's Document Ranking Performance in the Era of Transformers
Transformer structure has achieved great success in multiple applied machine learning communities, such as natural language processing (NLP), computer vision (CV) and information retrieval (IR). Transformer architecture's core mechanism -- attention requires $O(n^2)$ time complexity in training and $O(n)$ time complexity in inference. Many works have been proposed to improve the attention mechanism's scalability, such as Flash Attention and Multi-query Attention. A different line of work aims t…

This https://arxiv.org/abs/2310.05446 has been replaced.
link: https://scholar.google.com/scholar?q=a

RetSeg: Retention-based Colorectal Polyps Segmentation Network
Vision Transformers (ViTs) have revolutionized medical imaging analysis, showcasing superior efficacy compared to conventional Convolutional Neural Networks (CNNs) in vital tasks such as polyp classification, detection, and segmentation. Leveraging attention mechanisms to focus on specific image regions, ViTs exhibit contextual awareness in processing visual data, culminating in robust and precise predictions, even for intricate medical images. Moreover, the inherent self-attention mechanism in…

TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax
Tobias Christian Nauen, Sebastian Palacio, Andreas Dengel
https://arxiv.org/abs/2403.02920

TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax
The quadratic complexity of the attention mechanism represents one of the biggest hurdles for processing long sequences using Transformers. Current methods, relying on sparse representations or stateful recurrence, sacrifice token-to-token interactions, which ultimately leads to compromises in performance. This paper introduces TaylorShift, a novel reformulation of the Taylor softmax that enables computing full token-to-token interactions in linear time and space. We analytically determine the …

This https://arxiv.org/abs/2402.13573 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCV_…

ToDo: Token Downsampling for Efficient Generation of High-Resolution Images
Attention mechanism has been crucial for image diffusion models, however, their quadratic computational complexity limits the sizes of images we can process within reasonable time and memory constraints. This paper investigates the importance of dense attention in generative image models, which often contain redundant features, making them suitable for sparser attention mechanisms. We propose a novel training-free method ToDo that relies on token downsampling of key and value tokens to accelera…

This https://arxiv.org/abs/2403.14932 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

Attention-Driven Reasoning: Unlocking the Potential of Large Language Models
Large Language Models (LLMs) have shown remarkable capabilities, but their reasoning abilities and underlying mechanisms remain poorly understood. We present a novel approach to enhance LLMs' reasoning through attention mechanism optimization, without additional training data. We identify inefficiencies in the attention distribution caused by non-semantic tokens and propose an algorithm to re-balance the skewed distribution, enabling the model to abstract more nuanced knowledge. Our experiments…

🧠 Non-Intrusive Load Monitoring Based on Multiscale Attention Mechanisms
#ai

Non-Intrusive Load Monitoring Based on Multiscale Attention Mechanisms
With the development of smart grids and new power systems, the combination of non-intrusive load identification technology and smart home technology can provide users with the operating conditions of home appliances and equipment, thus reducing home energy loss and improving users’ ability to demand a response. This paper proposes a non-intrusive load decomposition model with a parallel multiscale attention mechanism (PMAM). The model can extract both local and global feature information and …

LiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification
Judy X Yang, Jun Zhou, Jing Wang, Hui Tian, Wee Chung Liew
https://arxiv.org/abs/2404.03883

LiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification
The fusion of hyperspectral and LiDAR data has been an active research topic. Existing fusion methods have ignored the high-dimensionality and redundancy challenges in hyperspectral images, despite that band selection methods have been intensively studied for hyperspectral image (HSI) processing. This paper addresses this significant gap by introducing a cross-attention mechanism from the transformer architecture for the selection of HSI bands guided by LiDAR data. LiDAR provides high-resolutio…

This https://arxiv.org/abs/2404.05101 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_qfi…

StockGPT: A GenAI Model for Stock Prediction and Trading
This paper introduces StockGPT, an autoregressive "number" model pretrained directly on the history of daily U.S. stock returns. Treating each return series as a sequence of tokens, the model excels at understanding and predicting the highly intricate stock return dynamics. Instead of relying on handcrafted trading patterns using historical stock prices, StockGPT automatically learns the hidden representations predictive of future returns via its attention mechanism. On a held-out test sample f…

This https://arxiv.org/abs/2403.17729 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csIR_…

EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention
To capture user preference, transformer models have been widely applied to model sequential user behavior data. The core of transformer architecture lies in the self-attention mechanism, which computes the pairwise attention scores in a sequence. Due to the permutation-equivariant nature, positional encoding is used to enhance the attention between token representations. In this setting, the pairwise attention scores can be derived by both semantic difference and positional difference. However,…

This https://arxiv.org/abs/2402.07422 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csAI_…

News Recommendation with Attention Mechanism
This paper explores the area of news recommendation, a key component of online information sharing. Initially, we provide a clear introduction to news recommendation, defining the core problem and summarizing current methods and notable recent algorithms. We then present our work on implementing the NRAM (News Recommendation with Attention Mechanism), an attention-based approach for news recommendation, and assess its effectiveness. Our evaluation shows that NRAM has the potential to significan…

Speech Emotion Recognition Via CNN-Transforemr and Multidimensional Attention Mechanism
Xiaoyu Tang, Yixin Lin, Ting Dang, Yuanfang Zhang, Jintao Cheng
https://arxiv.org/abs/2403.04743

Speech Emotion Recognition Via CNN-Transforemr and Multidimensional Attention Mechanism
Speech Emotion Recognition (SER) is crucial in human-machine interactions. Mainstream approaches utilize Convolutional Neural Networks or Recurrent Neural Networks to learn local energy feature representations of speech segments from speech information, but struggle with capturing global information such as the duration of energy in speech. Some use Transformers to capture global information, but there is room for improvement in terms of parameter count and performance. Furthermore, existing at…

Understanding the PULSAR Effect in Combined Radiotherapy and Immunotherapy through Attention Mechanisms with a Transformer Model
Hao Peng, Casey Moore, Debabrata Saha, Steve Jiang, Robert Timmerman
https://arxiv.org/abs/2403.04175

Understanding the PULSAR Effect in Combined Radiotherapy and Immunotherapy through Attention Mechanisms with a Transformer Model
PULSAR (personalized, ultra-fractionated stereotactic adaptive radiotherapy) is the adaptation of stereotactic ablative radiotherapy towards personalized cancer management. For the first time, we applied a transformer-based attention mechanism to investigate the underlying interactions between combined PULSAR and PD-L1 blockade immunotherapy based on a murine cancer model (Lewis Lung Carcinoma, LLC). The proposed approach is able to predict the trend of tumor volume change semi-quantitatively, …

This https://arxiv.org/abs/2309.01886 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_hepe…

Reconstruction of Unstable Heavy Particles Using Deep Symmetry-Preserving Attention Networks
Reconstructing unstable heavy particles requires sophisticated techniques to sift through the large number of possible permutations for assignment of detector objects to the underlying partons. Anapproach based on a generalized attention mechanism, symmetry preserving attention networks (SPA-NET), has been previously applied to top quark pair decays at the Large Hadron Collider which produce only hadronic jets. Here we extend the SPA-NET architecture to consider multiple input object types, suc…

DAM: A Universal Dual Attention Mechanism for Multimodal Timeseries Cryptocurrency Trend Forecasting
Yihang Fu, Mingyu Zhou, Luyao Zhang
https://arxiv.org/abs/2405.00522

DAM: A Universal Dual Attention Mechanism for Multimodal Timeseries Cryptocurrency Trend Forecasting
In the distributed systems landscape, Blockchain has catalyzed the rise of cryptocurrencies, merging enhanced security and decentralization with significant investment opportunities. Despite their potential, current research on cryptocurrency trend forecasting often falls short by simplistically merging sentiment data without fully considering the nuanced interplay between financial market dynamics and external sentiment influences. This paper presents a novel Dual Attention Mechanism (DAM) for…

QKFormer: Hierarchical Spiking Transformer using Q-K Attention
Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang, Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou, Yonghong Tian
https://arxiv.org/abs/2403.16552

QKFormer: Hierarchical Spiking Transformer using Q-K Attention
Spiking Transformers, which integrate Spiking Neural Networks (SNNs) with Transformer architectures, have attracted significant attention due to their potential for energy efficiency and high performance. However, existing models in this domain still suffer from suboptimal performance. We introduce several innovations to improve the performance: i) We propose a novel spike-form Q-K attention mechanism, tailored for SNNs, which efficiently models the importance of token or channel dimensions thr…

Continuous Sign Language Recognition Based on Motor attention mechanism and frame-level Self-distillation
Qidan Zhu, Jing Li, Fei Yuan, Quan Gan
https://arxiv.org/abs/2402.19118 <…

Continuous Sign Language Recognition Based on Motor attention mechanism and frame-level Self-distillation
Changes in facial expression, head movement, body movement and gesture movement are remarkable cues in sign language recognition, and most of the current continuous sign language recognition(CSLR) research methods mainly focus on static images in video sequences at the frame-level feature extraction stage, while ignoring the dynamic changes in the images. In this paper, we propose a novel motor attention mechanism to capture the distorted changes in local motion regions during sign language exp…

This https://arxiv.org/abs/2404.05843 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csLG_…

Softmax Attention with Constant Cost per Token
We propose a simple modification to the conventional attention mechanism applied by Transformers: Instead of quantifying pairwise query-key similarity with scaled dot-products, we quantify it with the logarithms of scaled dot-products of exponentials. Our modification linearizes attention with exponential kernel feature maps, whose corresponding feature function is infinite dimensional. We show that our modification is expressible as a composition of log-sums of exponentials, with a latent spac…

Implementing and Optimizing the Scaled Dot-Product Attention on Streaming Dataflow
Gina Sohn, Nathan Zhang, Kunle Olukotun
https://arxiv.org/abs/2404.16629

Implementing and Optimizing the Scaled Dot-Product Attention on Streaming Dataflow
Transformer models serve as the backbone of many state-ofthe-art language models, and most use the scaled dot-product attention (SDPA) mechanism to capture relationships between tokens. However, the straightforward implementation of SDPA has quadratic compute and memory complexity with respect to the sequence length. On processor architectures such as GPUs and TPUs, there is a robust body of prior work. However, little work has been performed on non-processor architectures.In this work, we show…

Incorporating Graph Attention Mechanism into Geometric Problem Solving Based on Deep Reinforcement Learning
Xiuqin Zhong, Shengyuan Yan, Gongqi Lin, Hongguang Fu, Liang Xu, Siwen Jiang, Lei Huang, Wei Fang
https://arxiv.org/abs/2403.14690

Incorporating Graph Attention Mechanism into Geometric Problem Solving Based on Deep Reinforcement Learning
In the context of online education, designing an automatic solver for geometric problems has been considered a crucial step towards general math Artificial Intelligence (AI), empowered by natural language understanding and traditional logical inference. In most instances, problems are addressed by adding auxiliary components such as lines or points. However, adding auxiliary components automatically is challenging due to the complexity in selecting suitable auxiliary components especially when …

This https://arxiv.org/abs/2312.16503 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csET_…

Attention-Enhanced Reservoir Computing
Photonic reservoir computing has been successfully utilized in time-series prediction as the need for hardware implementations has increased. Prediction of chaotic time series remains a significant challenge, an area where the conventional reservoir computing framework encounters limitations of prediction accuracy. We introduce an attention mechanism to the reservoir computing model in the output stage. This attention layer is designed to prioritize distinct features and temporal sequences, the…

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models
Arpit Aggarwal
https://arxiv.org/abs/2405.04585 https://

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models
There are several improvements proposed over the baseline Absolute Positional Encoding (APE) method used in original transformer. In this study, we aim to investigate the implications of inadequately representing positional encoding in higher dimensions on crucial aspects of the attention mechanism, the model's capacity to learn relative positional information, and the convergence of models, all stemming from the choice of sinusoidal basis functions. Through a combination of theoretical insight…

This https://arxiv.org/abs/2304.07235 has been replaced.
initial toot: https://mastoxiv.page/@arX…

Mapping of attention mechanisms to a generalized Potts model
Transformers are neural networks that revolutionized natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked language modeling (MLM). In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word. Despite the practical success of transformers, it remains unclear what type of data distribution self-attention can learn efficiently. Here, we …

MAC Advice for Facility Location Mechanism Design
Zohar Barak, Anupam Gupta, Inbal Talgam-Cohen
https://arxiv.org/abs/2403.12181 https://

MAC Advice for Facility Location Mechanism Design
Algorithms with predictions have attracted much attention in the last years across various domains, including variants of facility location, as a way to surpass traditional worst-case analyses. We study the $k$-facility location mechanism design problem, where the $n$ agents are strategic and might misreport their location. Unlike previous models, where predictions are for the $k$ optimal facility locations, we receive $n$ predictions for the locations of each of the agents. However, these pr…

This https://arxiv.org/abs/2405.04312 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCV_…

Inf-DiT: Upsampling Any-Resolution Image with Memory-Efficient Diffusion Transformer
Diffusion models have shown remarkable performance in image generation in recent years. However, due to a quadratic increase in memory during generating ultra-high-resolution images (e.g. 4096*4096), the resolution of generated images is often limited to 1024*1024. In this work. we propose a unidirectional block attention mechanism that can adaptively adjust the memory overhead during the inference process and handle global dependencies. Building on this module, we adopt the DiT structure for u…

This https://arxiv.org/abs/2302.10468 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCR_…

Soft Error Reliability Analysis of Vision Transformers
Vision Transformers (ViTs) that leverage self-attention mechanism have shown superior performance on many classical vision tasks compared to convolutional neural networks (CNNs) and gain increasing popularity recently. Existing ViTs works mainly optimize performance and accuracy, but ViTs reliability issues induced by soft errors in large-scale VLSI designs have generally been overlooked. In this work, we mainly study the reliability of ViTs and investigate the vulnerability from different arch…

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models
Arpit Aggarwal
https://arxiv.org/abs/2405.04585 https://

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models
There are several improvements proposed over the baseline Absolute Positional Encoding (APE) method used in original transformer. In this study, we aim to investigate the implications of inadequately representing positional encoding in higher dimensions on crucial aspects of the attention mechanism, the model's capacity to learn relative positional information, and the convergence of models, all stemming from the choice of sinusoidal basis functions. Through a combination of theoretical insight…

JefiAtten: An Attention Based Neural Network Model for Solving Maxwell's Equations with Charge and Current Sources
Ming-Yan Sun, Peng Xu, Jun-Jie Zhang, Tai-Jiao Du, Jian-Guo Wang
https://arxiv.org/abs/2402.16920

JefiAtten: An Attention Based Neural Network Model for Solving Maxwell's Equations with Charge and Current Sources
We present JefiAtten, a novel neural network model employing the attention mechanism to solve Maxwell's equations efficiently. JefiAtten uses self-attention and cross-attention modules to understand the interplay between charge density, current density, and electromagnetic fields. Our results indicate that JefiAtten can generalize well to a range of scenarios, maintaining accuracy across various spatial distribution and handling amplitude variations. The model showcases an improvement in comput…

This https://arxiv.org/abs/2110.07953 has been replaced.
link: https://scholar.google.com/scholar?q=a

Attention-based Estimation and Prediction of Human Intent to augment Haptic Glove aided Control of Robotic Hand
The letter focuses on Haptic Glove (HG) based control of a Robotic Hand (RH) executing in-hand manipulation of certain objects of interest. The high dimensional motion signals in HG and RH possess intrinsic variability of kinematics resulting in difficulty to establish a direct mapping of the motion signals from HG onto the RH. An estimation mechanism is proposed to quantify the motion signal acquired from the human controller in relation to the intended goal pose of the object being held by th…

This https://arxiv.org/abs/2402.03302 has been replaced.
link: https://scholar.google.com/scholar?q=a

Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining
Accurate medical image segmentation demands the integration of multi-scale information, spanning from local features to global dependencies. However, it is challenging for existing methods to model long-range global information, where convolutional neural networks (CNNs) are constrained by their local receptive fields, and vision transformers (ViTs) suffer from high quadratic complexity of their attention mechanism. Recently, Mamba-based models have gained great attention for their impressive a…

EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention
Zhen Tian, Wayne Xin Zhao, Changwang Zhang, Xin Zhao, Zhongrui Ma, Ji-Rong Wen
https://arxiv.org/abs/2403.17729

EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention
To capture user preference, transformer models have been widely applied to model sequential user behavior data. The core of transformer architecture lies in the self-attention mechanism, which computes the pairwise attention scores in a sequence. Due to the permutation-equivariant nature, positional encoding is used to enhance the attention between token representations. In this setting, the pairwise attention scores can be derived by both semantic difference and positional difference. However,…

This https://arxiv.org/abs/2305.11804 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_…

Powerful Radio-Loud Quasars are Triggered by Galaxy Mergers in the Cosmic Bright Ages
While supermassive black holes are ubiquitous features of galactic nuclei, only a small minority are observed during episodes of luminous accretion. The physical mechanism(s) driving the onset of fueling and ignition in these active galactic nuclei (AGN) are still largely unknown for many galaxies and AGN-selection criteria. Attention has focused on AGN triggering by means of major galaxy mergers gravitationally funneling gas towards the galactic center, with evidence both for and against this …

This https://arxiv.org/abs/2404.01183 has been replaced.
link: https://scholar.google.com/scholar?q=a

Positioning is All You Need
One can drive safely with a GPS without memorizing a world map (not to mention the dark regions that humans have never explored). Such a locality-based attention mechanism has a profound implication on our understanding of how the brain works. This paper refines the existing embodied cognition framework by turning the locality from a constraint to an advantage. Analogous to GPS-based navigation, positioning represents a computationally more efficient solution to flexible behaviors than reconstr…

The study of weak decays of doubly charmed baryons within rescattering mechanism
Xiao-Hui Hu, Cai Ping Jia, Fu Sheng Yu, Ye Xing
https://arxiv.org/abs/2403.09511

The study of weak decays of doubly charmed baryons within rescattering mechanism
The doubly charmed baryon $Ξ_{cc}^{++}$ has been observed by LHCb through the non-leptonic decay modes of $Ξ_{cc}^{++}\toΛ_{c}^{+}K^{-}π^{+}π^{+}$ and $Ξ_{c}^{+}π^{+}$ in 2017. After that, the experimentalists turn their attention to finding other doubly charmed baryons $Ξ_{cc}^{+}$ and $Ω_{cc}^{+}$. In this work, we investigate the nonleptonic weak decays of doubly charmed baryons ${\cal B}_{cc}\to{\cal B}_{c}P$, where ${\cal B}_{cc}$ denotes the doubly charmed baryons $(Ξ_{cc}^{++},…

This https://arxiv.org/abs/2309.16268 has been replaced.
initial toot: https://mastoxiv.page/@a…

Layer-dependent transport properties in the Moiré of strained homobilayer transition metal dichalcogenides
Bilayer moiré structures have attracted significant attention recently due to their spatially modulated layer degrees of freedom. However, the layer-dependent transport mechanism in the moiré structures is still a problem to be explored. Here we investigate the layer-dependent transport properties regulated by the strain, the interlayer bias and the number of moiré periods in a strained moiré homobilayer TMDs nanoribbon based on low-energy efficient models. The charge carriers can pass perf…

StockGPT: A GenAI Model for Stock Prediction and Trading
Dat Mai
https://arxiv.org/abs/2404.05101 https://arxiv.org/pdf/2404.05101

StockGPT: A GenAI Model for Stock Prediction and Trading
This paper introduces StockGPT, an autoregressive "number" model pretrained directly on the history of daily U.S. stock returns. Treating each return series as a sequence of tokens, the model excels at understanding and predicting the highly intricate stock return dynamics. Instead of relying on handcrafted trading patterns using historical stock prices, StockGPT automatically learns the hidden representations predictive of future returns via its attention mechanism. On a held-out test sample f…

DEFA: Efficient Deformable Attention Acceleration via Pruning-Assisted Grid-Sampling and Multi-Scale Parallel Processing
Yansong Xu, Dongxu Lyu, Zhenyu Li, Zilong Wang, Yuzhou Chen, Gang Wang, Zhican Wang, Haomin Li, Guanghui He
https://arxiv.org/abs/2403.10913

DEFA: Efficient Deformable Attention Acceleration via Pruning-Assisted Grid-Sampling and Multi-Scale Parallel Processing
Multi-scale deformable attention (MSDeformAttn) has emerged as a key mechanism in various vision tasks, demonstrating explicit superiority attributed to multi-scale grid-sampling. However, this newly introduced operator incurs irregular data access and enormous memory requirement, leading to severe PE underutilization. Meanwhile, existing approaches for attention acceleration cannot be directly applied to MSDeformAttn due to lack of support for this distinct procedure. Therefore, we propose a d…

This https://arxiv.org/abs/2308.10131 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_eco…

Agree to Disagree: Measuring Hidden Dissents in FOMC Meetings
Based on a record of dissents on FOMC votes and transcripts of the meetings from 1976 to 2017, we develop a deep learning model based on self-attention mechanism to create a measure of disagreement for members in each meeting. While dissents are rare, we find that members often have reservations with the policy decision, and the level of disagreement is mostly driven by current or predicted macroeconomic data. Using our model to evaluate speeches made by members between meetings, we find that t…

RankMamba, Benchmarking Mamba's Document Ranking Performance in the Era of Transformers
Zhichao Xu
https://arxiv.org/abs/2403.18276 https://

RankMamba, Benchmarking Mamba's Document Ranking Performance in the Era of Transformers
Transformer structure has achieved great success in multiple applied machine learning communities, such as natural language processing (NLP), computer vision (CV) and information retrieval (IR). Transformer architecture's core mechanism -- attention requires $O(n^2)$ time complexity in training and $O(n)$ time complexity in inference. Many works have been proposed to improve the attention mechanism's scalability, such as Flash Attention and Multi-query Attention. A different line of work aims t…

This https://arxiv.org/abs/2302.13451 has been replaced.
link: https://scholar.google.com/scholar?q=a

A low latency attention module for streaming self-supervised speech representation learning
The transformer is a fundamental building block in deep learning, and the attention mechanism is the transformer's core component. Self-supervised speech representation learning (SSRL) represents a popular use-case for the transformer architecture. Due to transformers' acausal behavior, the use of transformers for SSRL has been predominantly focused on acausal applications. However, several media processing problems, such as speech processing, require real-time solutions. In this paper, we pres…

How Transformers Learn Causal Structure with Gradient Descent
Eshaan Nichani, Alex Damian, Jason D. Lee
https://arxiv.org/abs/2402.14735 https://

How Transformers Learn Causal Structure with Gradient Descent
The incredible success of transformers on sequence modeling tasks can be largely attributed to the self-attention mechanism, which allows information to be transferred between different parts of a sequence. Self-attention allows transformers to encode causal structure which makes them particularly suitable for sequence modeling. However, the process by which transformers learn such causal structure via gradient-based training algorithms remains poorly understood. To better understand this proce…

#NutrientScience
1/7
In the realm of health and wellness, there's a powerful antioxidant that's been garnering attention for its myriad of benefits. Proanthocyanidines, often hailed as nature’s defense mechanism against oxidative stress, are not only fascinating but profoundly impactful on our well-being. Drawing upon the latest research, let's delve into the science behi…

This https://arxiv.org/abs/2402.07735 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Graph Structure Inference with BAM: Introducing the Bilinear Attention Mechanism
In statistics and machine learning, detecting dependencies in datasets is a central challenge. We propose a novel neural network model for supervised graph structure learning, i.e., the process of learning a mapping between observational data and their underlying dependence structure. The model is trained with variably shaped and coupled simulated input data and requires only a single forward pass through the trained network for inference. By leveraging structural equation models and employing …

RankMamba, Benchmarking Mamba's Document Ranking Performance in the Era of Transformers
Zhichao Xu
https://arxiv.org/abs/2403.18276 https://

RankMamba, Benchmarking Mamba's Document Ranking Performance in the Era of Transformers
Transformer structure has achieved great success in multiple applied machine learning communities, such as natural language processing (NLP), computer vision (CV) and information retrieval (IR). Transformer architecture's core mechanism -- attention requires $O(n^2)$ time complexity in training and $O(n)$ time complexity in inference. Many works have been proposed to improve the attention mechanism's scalability, such as Flash Attention and Multi-query Attention. A different line of work aims t…

Stampede Alert Clustering Algorithmic System Based on Tiny-Scale Strengthened DETR
Mingze Sun, Yiqing Wang, Zhenyi Zhao
https://arxiv.org/abs/2404.10359 ht…

Stampede Alert Clustering Algorithmic System Based on Tiny-Scale Strengthened DETR
A novel crowd stampede detection and prediction algorithm based on Deformable DETR is proposed to address the challenges of detecting a large number of small targets and target occlusion in crowded airport and train station environments. In terms of model design, the algorithm incorporates a multi-scale feature fusion module to enlarge the receptive field and enhance the detection capability of small targets. Furthermore, the deformable attention mechanism is improved to reduce missed detection…

This https://arxiv.org/abs/2305.11804 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_…

Powerful Radio-Loud Quasars are Triggered by Galaxy Mergers in the Cosmic Bright Ages
While supermassive black holes are ubiquitous features of galactic nuclei, only a small minority are observed during episodes of luminous accretion. The physical mechanism(s) driving the onset of fueling and ignition in these active galactic nuclei (AGN) are still largely unknown for many galaxies and AGN-selection criteria. Attention has focused on AGN triggering by means of major galaxy mergers gravitationally funneling gas towards the galactic center, with evidence both for and against this …

This https://arxiv.org/abs/2209.14145 has been replaced.
link: https://scholar.google.com/scholar?q=a

Multi-scale Attention Network for Single Image Super-Resolution
ConvNets can compete with transformers in high-level tasks by exploiting larger receptive fields. To unleash the potential of ConvNet in super-resolution, we propose a multi-scale attention network (MAN), by coupling classical multi-scale mechanism with emerging large kernel attention. In particular, we proposed multi-scale large kernel attention (MLKA) and gated spatial attention unit (GSAU). Through our MLKA, we modify large kernel attention with multi-scale and gate schemes to obtain the abu…

SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks
Xinyu Shi, Zecheng Hao, Zhaofei Yu
https://arxiv.org/abs/2403.14302 htt…

SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks
The remarkable success of Vision Transformers in Artificial Neural Networks (ANNs) has led to a growing interest in incorporating the self-attention mechanism and transformer-based architecture into Spiking Neural Networks (SNNs). While existing methods propose spiking self-attention mechanisms that are compatible with SNNs, they lack reasonable scaling methods, and the overall architectures proposed by these methods suffer from a bottleneck in effectively extracting local features. To address …

This https://arxiv.org/abs/2305.08396 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

MaxViT-UNet: Multi-Axis Attention for Medical Image Segmentation
Since their emergence, Convolutional Neural Networks (CNNs) have made significant strides in medical image analysis. However, the local nature of the convolution operator may pose a limitation for capturing global and long-range interactions in CNNs. Recently, Transformers have gained popularity in the computer vision community and also in medical image segmentation due to their ability to process global features effectively. The scalability issues of the self-attention mechanism and lack of th…

This https://arxiv.org/abs/2308.10131 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_eco…

Agree to Disagree: Measuring Hidden Dissents in FOMC Meetings
Based on a record of dissents on FOMC votes and transcripts of the meetings from 1976 to 2017, we develop a deep learning model based on self-attention mechanism to create a measure of disagreement for members in each meeting. While dissents are rare, we find that members often have reservations with the policy decision, and the level of disagreement is mostly driven by current or predicted macroeconomic data. Using our model to evaluate speeches made by members between meetings, we find that t…

Observation of intra-unit-cell superconductivity modulation
Tianheng Wei, Yanzhao Liu, Wei Ren, Ziqiang Wang, Jian Wang
https://arxiv.org/abs/2404.16683 ht…

Observation of intra-unit-cell superconductivity modulation
In unconventional high-temperature (high-Tc) superconductors, the symmetry-breaking electronic orders intertwined with the superconductivity provide important clues for understanding the nature of the unconventional pairing mechanism. Recently, an exotic superconducting order showing spatially periodic order parameter modulations and translational symmetry breaking, namely the pair density wave (PDW) state, has attracted broad attention. Without breaking translational symmetry, point group symm…

This https://arxiv.org/abs/2402.06919 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csIT_…

TREET: TRansfer Entropy Estimation via Transformer
Transfer entropy (TE) is a measurement in information theory that reveals the directional flow of information between processes, providing valuable insights for a wide range of real-world applications. This work proposes Transfer Entropy Estimation via Transformers (TREET), a novel transformer-based approach for estimating the TE for stationary processes. The proposed approach employs Donsker-Vardhan (DV) representation to TE and leverages the attention mechanism for the task of neural estimati…

This https://arxiv.org/abs/2305.08396 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

MaxViT-UNet: Multi-Axis Attention for Medical Image Segmentation
Since their emergence, Convolutional Neural Networks (CNNs) have made significant strides in medical image analysis. However, the local nature of the convolution operator may pose a limitation for capturing global and long-range interactions in CNNs. Recently, Transformers have gained popularity in the computer vision community and also in medical image segmentation due to their ability to process global features effectively. The scalability issues of the self-attention mechanism and lack of th…

This https://arxiv.org/abs/2402.10930 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csAR_…

ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters
The self-attention mechanism sets transformer-based large language model (LLM) apart from the convolutional and recurrent neural networks. Despite the performance improvement, achieving real-time LLM inference on silicon is challenging due to the extensively used Softmax in self-attention. Apart from the non-linearity, the low arithmetic intensity greatly reduces the processing parallelism, which becomes the bottleneck especially when dealing with a longer context. To address this challenge, we…

This https://arxiv.org/abs/2404.15848 has been replaced.
link: https://scholar.google.com/scholar?q=a

Detecting Conceptual Abstraction in LLMs
We present a novel approach to detecting noun abstraction within a large language model (LLM). Starting from a psychologically motivated set of noun pairs in taxonomic relationships, we instantiate surface patterns indicating hypernymy and analyze the attention matrices produced by BERT. We compare the results to two sets of counterfactuals and show that we can detect hypernymy in the abstraction mechanism, which cannot solely be related to the distributional similarity of noun pairs. Our findi…

Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting
Xiongxiao Xu, Yueqing Liang, Baixiang Huang, Zhiling Lan, Kai Shu
https://arxiv.org/abs/2404.14757 <…

Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting
Time series forecasting is an important problem and plays a key role in a variety of applications including weather forecasting, stock market, and scientific simulations. Although transformers have proven to be effective in capturing dependency, its quadratic complexity of attention mechanism prevents its further adoption in long-range time series forecasting, thus limiting them attend to short-range range. Recent progress on state space models (SSMs) have shown impressive performance on modeli…

This https://arxiv.org/abs/2403.18257 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
Transformers have been the most successful architecture for various speech modeling tasks, including speech separation. However, the self-attention mechanism in transformers with quadratic complexity is inefficient in computation and memory. Recent models incorporate new layers and modules along with transformers for better performance but also introduce extra model complexity. In this work, we replace transformers with Mamba, a selective state space model, for speech separation. We propose dua…

This https://arxiv.org/abs/2308.09419 has been replaced.
link: https://scholar.google.com/scholar?q=a

Attention Calibration for Transformer-based Sequential Recommendation
Transformer-based sequential recommendation (SR) has been booming in recent years, with the self-attention mechanism as its key component. Self-attention has been widely believed to be able to effectively select those informative and relevant items from a sequence of interacted items for next-item prediction via learning larger attention weights for these items. However, this may not always be true in reality. Our empirical analysis of some representative Transformer-based SR models reveals tha…

ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters
Shiwei Liu, Guanchen Tao, Yifei Zou, Derek Chow, Zichen Fan, Kauna Lei, Dennis Sylvester, Gregory Kielian, Mehdi Saligane
https://arxiv.org/abs/2402.10930

ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters
The self-attention mechanism sets transformer-based large language model (LLM) apart from the convolutional and recurrent neural networks. Despite the performance improvement, achieving real-time LLM inference on silicon is challenging due to the extensively used Softmax in self-attention. Apart from the non-linearity, the low arithmetic intensity greatly reduces the processing parallelism, which becomes the bottleneck especially when dealing with a longer context. To address this challenge, we…

This https://arxiv.org/abs/2308.09419 has been replaced.
link: https://scholar.google.com/scholar?q=a

Attention Calibration for Transformer-based Sequential Recommendation
Transformer-based sequential recommendation (SR) has been booming in recent years, with the self-attention mechanism as its key component. Self-attention has been widely believed to be able to effectively select those informative and relevant items from a sequence of interacted items for next-item prediction via learning larger attention weights for these items. However, this may not always be true in reality. Our empirical analysis of some representative Transformer-based SR models reveals tha…

Dyakonov-Perel-like Orbital and Spin Relaxations in Centrosymmetric Systems
Jeonghun Sohn, Jongjun M. Lee, Hyun-Woo Lee
https://arxiv.org/abs/2404.11780 ht…

Dyakonov-Perel-like Orbital and Spin Relaxations in Centrosymmetric Systems
The Dyakonov-Perel (DP) mechanism of spin relaxation has long been considered irrelevant in centrosymmetric systems since it was developed originally for non-centrosymmetric ones. We investigate whether this conventional understanding extends to the realm of orbital relaxation, which has recently attracted significant attention. Surprisingly, we find that orbital relaxation in centrosymmetric systems exhibits the DP-like behavior in the weak scattering regime. Moreover, the DP-like orbital rela…

This https://arxiv.org/abs/2403.11505 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Covid-19 detection from CT scans using EfficientNet and Attention mechanism
Manual diagnosis and analysis of COVID-19 through the examination of lung Computed Tomography (CT) scan images by physicians tends to result in inefficiency, especially with high patient volumes and numerous images per patient. We address the need for automation by developing a deep learning model-based pipeline for COVID-19 detection from CT scan images of the lungs. The Domain adaptation, Explainability, and Fairness in AI for Medical Image Analysis Workshop and COVID-19 Diagnosis Competition…

gTBLS: Generating Tables from Text by Conditional Question Answering
Anirudh Sundar, Christopher Richardson, Larry Heck
https://arxiv.org/abs/2403.14457 ht…

gTBLS: Generating Tables from Text by Conditional Question Answering
Distilling large, unstructured text into a structured, condensed form such as tables is an open research problem. One of the primary challenges in automatically generating tables is ensuring their syntactic validity. Prior approaches address this challenge by including additional parameters in the Transformer's attention mechanism to attend to specific rows and column headers. In contrast to this single-stage method, this paper presents a two-stage approach called Generative Tables (gTBLS). The…

This https://arxiv.org/abs/2403.11505 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Covid-19 detection from CT scans using EfficientNet and Attention mechanism
Manual diagnosis and analysis of COVID-19 through the examination of lung Computed Tomography (CT) scan images by physicians tends to result in inefficiency, especially with high patient volumes and numerous images per patient. We address the need for automation by developing a deep learning model-based pipeline for COVID-19 detection from CT scan images of the lungs. The Domain adaptation, Explainability, and Fairness in AI for Medical Image Analysis Workshop and COVID-19 Diagnosis Competition…

Allspark: Workload Orchestration for Visual Transformers on Processing In-Memory Systems
Mengke Ge, Junpeng Wang, Binhan Chen, Yingjian Zhong, Haitao Du, Song Chen, Yi Kang
https://arxiv.org/abs/2403.15069

Allspark: Workload Orchestration for Visual Transformers on Processing In-Memory Systems
The advent of Transformers has revolutionized computer vision, offering a powerful alternative to convolutional neural networks (CNNs), especially with the local attention mechanism that excels at capturing local structures within the input and achieve state-of-the-art performance. Processing in-memory (PIM) architecture offers extensive parallelism, low data movement costs, and scalable memory bandwidth, making it a promising solution to accelerate Transformer with memory-intensive operations.…

Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
Xilin Jiang, Cong Han, Nima Mesgarani
https://arxiv.org/abs/2403.18257

Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
Transformers have been the most successful architecture for various speech modeling tasks, including speech separation. However, the self-attention mechanism in transformers with quadratic complexity is inefficient in computation and memory. Recent models incorporate new layers and modules along with transformers for better performance but also introduce extra model complexity. In this work, we replace transformers with Mamba, a selective state space model, for speech separation. We propose dua…

This https://arxiv.org/abs/2402.03835 has been replaced.
link: https://scholar.google.com/scholar?q=a

Transformer based Endmember Fusion with Spatial Context for Hyperspectral Unmixing
In recent years, transformer-based deep learning networks have gained popularity in Hyperspectral (HS) unmixing applications due to their superior performance. The attention mechanism within transformers facilitates input-dependent weighting and enhances contextual awareness during training. Drawing inspiration from this, we propose a novel attention based Hyperspectral Unmixing algorithm called Transformer based Endmember Fusion with Spatial Context for Hyperspectral Unmixing (FusionNet). This…

Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
Xilin Jiang, Cong Han, Nima Mesgarani
https://arxiv.org/abs/2403.18257

Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
Transformers have been the most successful architecture for various speech modeling tasks, including speech separation. However, the self-attention mechanism in transformers with quadratic complexity is inefficient in computation and memory. Recent models incorporate new layers and modules along with transformers for better performance but also introduce extra model complexity. In this work, we replace transformers with Mamba, a selective state space model, for speech separation. We propose dua…

A Mutual Inclusion Mechanism for Precise Boundary Segmentation in Medical Images
Yizhi Pan, Junyi Xin, Tianhua Yang, Teeradaj Racharak, Le-Minh Nguyen, Guanqun Sun
https://arxiv.org/abs/2404.08201

A Mutual Inclusion Mechanism for Precise Boundary Segmentation in Medical Images
In medical imaging, accurate image segmentation is crucial for quantifying diseases, assessing prognosis, and evaluating treatment outcomes. However, existing methods lack an in-depth integration of global and local features, failing to pay special attention to abnormal regions and boundary details in medical images. To this end, we present a novel deep learning-based approach, MIPC-Net, for precise boundary segmentation in medical images. Our approach, inspired by radiologists' working pattern…

This https://arxiv.org/abs/2404.03883 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

LiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification
The fusion of hyperspectral and LiDAR data has been an active research topic. Existing fusion methods have ignored the high-dimensionality and redundancy challenges in hyperspectral images, despite that band selection methods have been intensively studied for hyperspectral image (HSI) processing. This paper addresses this significant gap by introducing a cross-attention mechanism from the transformer architecture for the selection of HSI bands guided by LiDAR data. LiDAR provides high-resolutio…